Survey data processing device, survey data processing method, and program therefor

ABSTRACT

A technique for making multiple still images, which are photographed by a traveling mobile body, and the traveled route of the photographing correspond to each other, is obtained. A survey data processing device includes an input unit  101,  an image processing unit  102,  and a synchronous processing unit  103.  The input unit  101  is configured to receive image data of multiple still images, which are photographed from a mobile body flying, and receive flight data, in which a flight route of the mobile body is measured. The image processing unit  102  is configured to estimate a flight route of the mobile body based on changes in positions of feature points included in the multiple still images on a screen. The synchronous processing unit  103  is configured to specify a matching relationship between the flight route of the flight data and the estimated flight route.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to a surveying technique.

2. Background Art

For example, a technique for obtaining a three-dimensional model of thetopography of an object based on image data (three-dimensionaltopographic data in which the topography of the object is modeled aselectronic data) is publicly known (for example, refer to JapaneseUnexamined Patent Application Laid-Open No. 2013-96745). The image datamay be obtained by photographing a civil engineering worksite or thelike from the air. In such a technique, information of photographinglocations must be linked with the photographing data.

In the above technique, a method of using an autonomously flyingunmanned air vehicle (UAV) equipped with a camera may be used. In thiscase, the clock included in the camera is synchronized with a GNSS unit(location identifying unit using a navigation satellite) and an IMU(inertial navigation unit), which are mounted on the UAV. Then, arelationship between an obtained still image (frame image in the case ofusing a moving image) and a location and an attitude of the camera (UAV)is obtained.

SUMMARY OF THE INVENTION

However, considering the costs, the weights mountable on a UAV, and theamounts of electricity consumed, there may be cases of using hardware inwhich photographed still images and the flight route cannot besynchronized with each other. In such cases, processing for making theobtained image data and the flight route correspond to each other isnecessary. This processing is conventionally performed by manualoperation, but it is complicated to perform this processing on each ofseveral hundred or more still images, and therefore, improvements aredesired. In view of these circumstances, an object of the presentinvention is to provide a technique for making multiple still images,which are photographed by a camera that is traveling, and the traveledroute of the photographing correspond to each other.

A first aspect of the present invention provides a survey dataprocessing device including an input unit, a flight route estimatingunit, and a matching relationship specifying unit. The input unitreceives image data of multiple still images, which are photographedfrom a mobile body flying, and receives flight data, in which a flightroute of the mobile body is measured. The flight route estimating unitestimates a flight route of the mobile body based on changes inpositions of feature points included in the multiple still images on ascreen. The matching relationship specifying unit specifies a matchingrelationship between the flight route of the flight data and theestimated flight route.

When photographing is performed consecutively from a mobile body, andthe obtained still images are arranged in the time sequence order,photographed objects seem to move gradually between the frames of thestill images. This is because the photographed area is gradually shiftedin accordance with the movement of the camera. This is the sameprinciple as in the case in which scenery outside a train window appearsto move rearward. Here, considering a specific feature point included inthe image, when the movement of the specific feature point is visuallytracked, it corresponds to the movement of the camera. Therefore, byanalyzing the movement of a feature point in the image, a relativemovement route of the camera can be estimated. In this method, since atrue scale is not provided, the estimated route is a relative model. Inview of this, in the present invention, the estimated route and atraveled route obtained in a mobile body (obtained by a GNSS unit, forexample) are compared with each other so as to match trajectoriesthereof, whereby a matching relationship therebetween is obtained. Afterthe two routes are made to coincide with each other (or have arelationship which can be recognized as correspondence), a relationshipbetween the traveled route and the obtained still images is determined.That is, a matching relationship between the multiple still images,which are obtained by photographing by a camera that is traveling, andthe traveled route of the photographing, is determined.

According to a second aspect of the present invention, in the firstaspect of the present invention, the flight route estimating unit mayperform a processing for obtaining a moving-direction vector, whichcharacterizes a movement of the feature point on the screen, and aprocessing for estimating the flight route based on the moving-directionvector.

According to a third aspect of the present invention, in one of thefirst and the second aspects of the present invention, the matchingrelationship specifying unit may calculate a location of the estimatedflight route so that a total of distances between vectors indicating theestimated flight route and vectors indicating the flight route of theflight data be minimum.

A fourth aspect of the present invention provides a survey dataprocessing method including: receiving image data of multiple stillimages, which are photographed from a mobile body flying, and flightdata, in which a flight route of the mobile body is measured, estimatinga flight route of the mobile body based on changes in positions offeature points included in the multiple still images on a screen, andspecifying a matching relationship between the flight route of theflight data and the estimated flight route.

A fifth aspect of the present invention provides a computer programproduct including a non-transitory computer-readable medium storingcomputer-executable program codes. The computer-executable program codesinclude program code instructions for: receiving image data of multiplestill images, which are photographed from a mobile body flying, andflight data, in which a flight route of the mobile body is measured,estimating a flight route of the mobile body based on changes inpositions of feature points included in the multiple still images on ascreen, and specifying a matching relationship between the flight routeof the flight data and the estimated flight route.

According to the present invention, a technique for making multiplestill images, which are obtained by photographing by a camera that istraveling, and the traveled route of the photographing correspond toeach other is obtained.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an embodiment.

FIG. 2 is a block diagram of an image processing unit.

FIG. 3 is a block diagram of a UI controlling unit.

FIG. 4 is an explanatory diagram showing a principle of templatematching.

FIG. 5 is an explanatory diagram showing a principle of a processing forestimating a flight route from images.

FIG. 6 is a flow chart showing an example of a processing procedure.

FIG. 7 is a flow chart showing an example of a processing procedure.

FIG. 8 is a view showing a relationship between still images and atraveled route.

FIG. 9 is a view showing an example of a GUI image displayed on ascreen.

FIG. 10 is a view showing an example of a GUI image displayed on ascreen.

FIG. 11 is a view showing an example of a GUI image displayed on ascreen.

FIG. 12 is a view showing an example of a GUI image displayed on ascreen.

FIG. 13 is a view showing an example of a GUI image displayed on ascreen.

FIG. 14 is a view showing an example of a GUI image displayed on ascreen.

PREFERRED EMBODIMENTS OF THE INVENTION Outline

In this embodiment, an autonomously flying unmanned air vehicle (UAV)with a piece of equipment mounted with a camera is used. The UAV isequipped with a GNSS unit (location identifying unit using a navigationsatellite) and an IMU (inertial navigation unit), and it is capable ofautonomous flight. The UAV consecutively photographs a ground surface bythe camera while flying. Specifically, the UAV consecutively performsprocessing of photographing a first still image at time t1, a secondstill image at time t2, and a third still image at time t3 while flying.The interval of the photographing is determined as needed, and forexample, it may be 2 seconds. Alternatively, a moving image may bephotographed, and frame images constructing the moving image may be usedas still images. That is, a moving image is constructed of multipleframe images that are aligned on a time axis, such as of a first frameimage photographed at time t1, a second frame image photographed at timet2, and so on, and therefore, the frame images may be used as stillimages in this embodiment.

In the photographing, since the photographing is performed while flying,the positions of viewpoints shift little by little, whereby numerousstill images including photographed areas that slightly differ from eachother are obtained. The camera is equipped with a clock and records thephotographing time, whereby image data of photographed still images,which are respectively linked with its photographing time, is obtained.The UAV has a function of recording data relating to the flight routethat is measured by the GNSS unit and the IMU.

In this embodiment, the UAV records location information of the flightroute at an interval of 0.2 seconds by using the GNSS. The datarecording the flight route can be output from the UAV to the outside aslog data after the flight. Meanwhile, the camera photographs stillimages at an interval of 2 seconds and stores the image data. The UAVand the camera are respectively equipped with a clock, but the clocksthereof are not synchronized with each other. Therefore, when the flightis completed, immediately after the image data, and the log datarelating to the flight route, are respectively obtained from the cameraand the UAV, the image data and the log data relating to the flightroute do not correspond to each other. This embodiment relates to atechnique of making the image data and the flight route correspond toeach other by software processing.

Structure

FIG. 1 shows a survey data processing device 100 for performing aprocessing for calculating the relationship between the image dataobtained from the camera and the flight route data obtained from theUAV. The survey data processing device 100 shown in FIG. 1 functions asa computer and includes a CPU, a solid electronic memory, a hard diskstorage unit, various types of interfaces, and other arithmetic elementsas necessary. FIG. 1 shows each kind of the functioning units, which areunderstood as functions. One or more of each kind of the functioningunits shown in FIG. 1 may be constructed of software or may beconstructed of dedicated hardware.

For example, the survey data processing device 100 may be constructed ofdedicated hardware, or the functions of the functioning units shown inFIG. 1 may be performed by software by using a general purpose computer.In the case of using a general purpose computer, images (describedlater) are displayed on a display provided to or connected to thecomputer, and the operator performs various kinds of operations by usinga UI (User Interface) that can be used in the computer. In addition, atleast some of the functions of the survey data processing device 100 maybe performed by a tablet computer (tablet terminal) or a smartphone.

The survey data processing device 100 includes an input unit 101, animage processing unit 102, a synchronous processing unit 103, a display104, a UI (User Interface) controlling unit 105, a storing unit 106, andan output unit 107.

Image data of images, which are photographed by the camera mounted onthe UAV while flying, log data (data of flight route recorded) of theUAV, and data relating to a tilt of the UAV measured by the IMU, areinput to the input unit 101. The image processing unit 102 performs aprocessing for estimating the flight route of the UAV based on multiplestill images, which are photographed by the UAV while flying. FIG. 2shows a block diagram of the image processing unit 102. The imageprocessing unit 102 includes a feature point extracting unit 111, amatching point identifying unit 112, a moving-direction vector featureextracting unit 113, a flight route estimating unit 114, and anormalizing processing unit 115.

The feature point extracting unit 111 extracts feature points from thestill images. As the feature points, points that can be differentiatedfrom surroundings, for example, edge portions and portions having colorsthat are different from surroundings, are extracted. The extraction ofthe feature points is performed by software processing using adifferential filter such as a Sobel filter, a Laplacian filter, aPrewitt filter, a Roberts filter, or the like.

The matching point identifying unit 112 identifies matchingrelationships between the feature points, which are extractedrespectively from two still images. That is, the matching pointidentifying unit 112 performs a processing for identifying featurepoints, which are extracted from one still image, with feature points inanother still image. This processing for identifying the matchingrelationships of the feature points is performed by using the templatematching shown in FIG. 4, for example.

As the template matching, a SSDA method (Sequential Similarity DetectionAlgorithm), a cross-correlation coefficient method, or the like may beused. An example of the template matching will be described below. Thetemplate matching is a method in which coordinate data of images in twocoordinate systems are compared with each other and a matchingrelationship between the two images is calculated by correlationrelationship between the coordinate data. In the template matching, thematching relationship between feature points of two images seen fromdifferent viewpoints is calculated. FIG. 4 is an explanatory diagram forexplaining the principle of the template matching. In this method, asshown in FIG. 4, a template image of N₁×N₁ pixels is moved on a searchrange (M₁−N₁+1)² within an input image of M₁×M₁ pixels which is largerthan the template image, and an upper left position of the templateimage is calculated so that the cross-correlation function C(a, b)denoted by the following First Formula represent the maximum value (thatis, the correlation degree becomes maximum).

$\begin{matrix}{{{First}\mspace{14mu} {Formula}}\mspace{610mu} {{C\left( {a,b} \right)} = {\sum\limits_{m_{1} = 0}^{N_{1} - 1}{\sum\limits_{n_{1} = 0}^{N_{1} - 1}\frac{\left\{ {{I_{({a,b})}\left( {m_{1},n_{1}} \right)} - 1} \right\} \left\{ {{T\left( {m_{1},n_{1}} \right)} - T} \right\}}{\sqrt{I_{\sigma \; {ab}}T_{\sigma}}}}}}} \\{{{Here},{\overset{\_}{I} = {\frac{1}{N_{1}^{2}}{\sum\limits_{m_{1} = 0}^{N_{1} - 1}{\sum\limits_{n_{1} = 0}^{N_{1} - 1}{I_{({a,b})}\left( {m_{1},n_{1}} \right)}}}}}}{\overset{\_}{T} = {\frac{1}{N_{1}^{2}}{\sum\limits_{m_{1} = 0}^{N_{1} - 1}{\sum\limits_{n_{1} = 0}^{N_{1} - 1}{T\left( {m_{1},n_{1}} \right)}}}}}{I_{\sigma \; {ab}} = {\frac{1}{N_{1}^{2}}{\sum\limits_{m_{1} = 0}^{N_{1} - 1}{\sum\limits_{n_{1} = 0}^{N_{1} - 1}\left\{ {{I_{({a,b})}\left( {m_{1},n_{1}} \right)} - I} \right\}^{2}}}}}{T_{\sigma} = {\frac{1}{N_{1}^{2}}{\sum\limits_{m_{1} = 0}^{N_{1} - 1}{\sum\limits_{n_{1} = 0}^{N_{1} - 1}\left\{ {{T\left( {m_{1},n_{1}} \right)} - T} \right\}^{2}}}}}{{I_{({a,b})}\left( {m_{1},n_{1}} \right)}\text{:}\mspace{11mu} {Local}\mspace{14mu} {image}\mspace{14mu} {of}\mspace{14mu} {input}\mspace{14mu} {image}}{{T\left( {m_{1},n_{1}} \right)}\text{:}\mspace{11mu} {Template}\mspace{14mu} {image}}}\end{matrix}$

The above processing is performed by changing the magnification of theone image and rotating the one image. In a condition in which thecorrelation degree is the maximum, the matched region of both images iscalculated, and feature points in this region are extracted, wherebymatching points are detected.

By using the template matching, a portion that matches between twocompared images can be identified, and the matching relationship betweenthe two images can be calculated. In this method, the relativepositional relationship between the two images is calculated so that thedegree of the correlation relationship between the two images can be themaximum. The correlation relationship between the two images iscalculated based on the feature points of the two images.

The moving-direction vector feature extracting unit 113 extractsfeatures of vectors correlating the movement of the UAV (camera), basedon changes in the still images, which are generated due to the elapse oftime. The vectors are called “moving-direction vectors”. By calculatingtrajectories of the moving-direction vectors in accordance with theelapse of time, a relative flight route of the UAV is calculated.

The processing performed by the moving-direction vector featureextracting unit 113 will be described hereinafter. First, to providebackground information, a relationship between a still image obtained ateach time and an actual flight route is described. FIG. 8 showscorresponding relationships between still images shifted and movementvectors obtained from the log data of the UAV. FIG. 8 shows consecutivestill images Nos. 1 to 4, showing conditions in which the photographedlocation is slightly shifted every moment as the UAV flies. The flightroute (changes in the movement vectors) obtained from the log data ofthe UAV is shown under the consecutive still images Nos. 1 to 4.

As shown in FIG. 8, changes in the still images correspond to themovement in the flight route. By using this phenomenon, themoving-direction vector feature extracting unit 113 performs aprocessing for obtaining moving-direction vectors, which characterizemovement in the still images relative to the camera, by analyzing thestill images.

FIG. 5 schematically shows still images, which are respectively obtainedat times t1, t2, and t3, and the times t1, t2, and t3 passed in thisorder. The still images shown in FIG. 5 show a simplified crossroad witha building standing on a corner. FIG. 5 shows a case in which a featurepoint X moved leftward from time t1 to time t2 and then further moved toa lower left side from time t2 to time t3.

In this case, the moving-direction vector feature extracting unit 113focuses on the feature point X and extracts changes in the locationthereof. For example, in this case, by comparing the still image at thetime t1 with the still image at the time t2, a moving-direction vector Ais obtained as a vector indicating the moving direction and the movingdistance of the feature point X. Then, by comparing the still image atthe time t2 with the still image at the time t3, a moving-directionvector B is obtained as a vector indicating the moving direction and themoving distance of the feature point X. Thereafter, the trajectories ofthe moving-direction vectors A and B are connected with each other,whereby a route C is obtained. Thus, by comparing each still image witha next still image that is photographed in a consecutive manner,moving-direction vectors (for example, the moving-direction vectors Aand B shown in FIG. 5) for characterizing changes in the still images isobtained.

The flight route estimating unit 114 estimates the flight route of theUAV (traveled route of the camera) based on the moving-directionvectors. For example, considering the route C shown in FIG. 5, the UAVmoves along a route that is the reverse of the route indicated by themoving-direction vectors A and B relative to the ground surface.Therefore, the flight route of the UAV from the times t1 to t3 isindicated as shown by the flight route L. Although the above isdescribed in the case of three instances of the times t1, t2, and t3,the trajectory of the flight route of the UAV is estimated by furtherperforming the same processing on still images that are obtained atmultiple times. The above processing is performed by the flight routeestimating unit 114.

The estimated flight trajectory obtained at this stage does not have atrue scale, and therefore, it is relative. That is, in the flight routeobtained in accordance with the principle shown in FIG. 5, data relatingto the flight distance is not included.

The normalizing processing unit 115 shown in FIG. 2 enlarges and reducesthe estimated flight trajectory of the UAV, which is estimated by theflight route estimating unit 114, so as to adjust the difference in thescale from the actual flight route of the UAV, which is obtained fromthe log data of the UAV. That is, since the estimated flight route ofthe UAV obtained by the flight route estimating unit 114 is a relativeflight trajectory that is not provided with a true scale, the scale isadjusted as described above so as to be made to correspond to the scaleof the actual flight route obtained from the log data.

In general, flight of a UAV may be conducted along a route making atrajectory as shown in FIG. 9. Therefore, the estimated flighttrajectory is enlarged or reduced so as to have a scale that isapproximately the same as the scale of the actual trajectory as shown inFIG. 9. Thus, a trajectory of the estimated flight route, of which thescale is adjusted so as to be approximately the same as the true scale,is obtained. The above is the processing performed by the normalizingprocessing unit 115.

Then, to return to FIG. 1, the synchronous processing unit 103 performsa processing for comparing the data of the estimated flight route of theUAV, which is estimated by the image processing unit 102, with the dataof the flight route (log data), which is actually recorded by the UAV,and synchronizing (matching) these data with each other.

A specific example of the processing performed by the synchronousprocessing unit 103 is described below. The synchronous processing unit103 matches a trajectory of a flight route A of the UAV, which isestimated by the image processing unit 102, with a flight route B, whichis actually recorded, by rotating, parallel moving, enlarging, orreducing the flight route A. By performing the matching, the matchingrelationship between the flight routes A and B is clarified, and whichstill image was photographed at a specific position in the flight routeis identified.

Specifically, since the relationship between the location and the timeis determined in the flight route B, when the flight route A and theflight route B are matched with each other, the flight route A (flightroute that is estimated by the image analysis) is provided with a truescale, and the relationship between the location and the time isclarified in the flight route A. Then, by clarifying the relationshipbetween the location and the time in the flight route A, information ofthe photographing times and the photographing locations of the stillimages for the flight route A is obtained. That is, the photographingtime and the photographing location of a specific still image areidentified in the image data that is obtained by the camera. Theidentified data of the photographing time and the photographinglocation, and the data relating to the tilt of the UAV at the time whenthe UAV photographed a corresponding image, are added to each image fileas header information (for example, a geotag).

The display 104 displays various GUI (Graphical User Interface) imagesas shown in FIGS. 9 to 14. The display 104 also performs displays, thatare necessary for controlling the survey data processing device 100, anddisplays of various kinds of data. When a notebook computer is used asthe survey data processing device 100, the display of the notebookcomputer may be used as the display 104.

The UI controlling unit 105 controls functions of GUI as shown in FIGS.9 to 14. FIG. 3 shows a block diagram of the UI controlling unit 105.The UI controlling unit 105 includes a route point selection receivingunit 121, a corresponding still image obtaining unit 122, amoving-direction indication embedding unit 123, and a displaycontrolling unit 124 for displaying a moving-direction indicationembedded still image.

The route point selection receiving unit 121 receives the contents whenthe operator selects a specific point (route point) of the flight routethat is obtained from the log data. For example, FIG. 10 shows anexample of a GUI image displayed on the display 104. A flight routeobtained from the log data is shown on the right side in FIG. 10, and astill image obtained from image data is shown on the left side in FIG.10. Here, the still image shown on the left side was photographed at aposition at a specific point (marker point with a cloverleaf shape) ofthe flight route on the right side. The position of the marker with thecloverleaf shape can be selected by controlling the GUI by the operator.The contents of the operation are received by the route point selectionreceiving unit 121.

The corresponding still image obtaining unit 122 obtains a still image(still image photographed by the camera), which corresponds to thespecific position of the flight route that is received by the routepoint selection receiving unit 121, from the image data. Themoving-direction indication embedding unit 123 performs a processing forembedding a mark (in the case shown in FIG. 10, an arrow) in the stillimage that is obtained by the corresponding still image obtaining unit122, for indicating the moving direction of the UAV (camera). The movingdirection of the UAV is calculated by the principle shown in FIG. 5.That is, by comparing still images with each other, which arephotographed in a consecutive manner, relative movements of featurepoints between the still images are detected, and the moving directionand the moving distance of the UAV are calculated therefrom. By thisprocessing, the moving direction and the moving distance of the UAV arecalculated based on a starting point in a preceding still image, andthey are displayed by an arrow (vector display) in a next still image.This processing is performed by the moving-direction indicationembedding unit 123.

The display controlling unit 124 performs controlling for displaying thestill image, in which a mark for indicating the moving direction of theUAV is embedded, on the display 104. An example of a still image, whichis displayed on the display 104 by the display controlling unit 124, isshown on the left side of FIG. 10.

Then, to return to FIG. 1, the storing unit 106 stores various data,which are necessary for operating the survey data processing device 100,and various data, which are obtained as a result of the operation. Forexample, the storing unit 106 stores image data obtained by the camera,the log data of the UAV, various kinds of programs for operating thesurvey data processing device 100, intermediate data to be processed bythe survey data processing device 100, and data obtained as a result ofthe processing. The output unit 107 outputs various data, which areobtained by the survey data processing device 100, and various data,which are used in the survey data processing device 100, to the outside.

Example of Processing

FIG. 6 shows an example of a processing procedure. When the processingis started, image data is input (step S101), and the log data is alsoinput (step S102). After the image data is input, feature points areextracted in each of the still images (step S103). The processing ofthis step is performed by the feature point extracting unit 111 shown inFIG. 2. After the feature points are extracted, the feature points,which match between still images that are photographed in a consecutivemanner, are identified (step S104). The processing of this step isperformed by the matching point identifying unit 112.

After the matching points are identified, features of moving-directionvectors are extracted by using the method exemplified in FIG. 5 (stepS105). The processing of this step is performed by the moving-directionvector feature extracting unit moving direction vector featureextracting unit 113 shown in FIG. 2. Next, the procedure advances tostep S106, and a flight route of the UAV is calculated based on themoving-direction vectors in accordance with the principle shown in thelower side in FIG. 5. The processing of this step is performed by theflight route estimating unit 114 shown in FIG. 2.

After the flight route is estimated, a normalizing processing (stepS107) is performed, whereby the scale of the flight route, which isobtained from the log data 102, and the scale of the flight route, whichis estimated in step S106, are made to correspond with each other. Then,the estimated flight route is in parallel moved, rotated, enlarged, orreduced, so as to match with the flight route obtained from the log data102 (step S108). The matching is performed by the least-squares method.

In the processing of this step, the location of the estimated flightroute is calculated by the least-squares method so that the total ofdistances between the vectors for indicating the estimated flight route,which is obtained by the method shown in FIG. 5, and the vectors forindicating the flight route, which is obtained from the log data, willbe minimum. Naturally, the estimated flight route and the flight routeobtained from the log data may be matched with each other by anothermethod. By performing step S108, survey data, in which the photographedmultiple still images are linked with the flight route that is obtainedby the UAV, is obtained. The survey data is stored in the storing unit106 shown in FIG. 1.

By matching the flight route, which is obtained from the log data 102,with the flight route, which is estimated in step S106, thecorresponding relationship between the flight route and the photographedstill images is determined. As a result, the GUI images as exemplifiedin FIGS. 9 to 14 can be displayed (step S109).

FIGS. 9 to 14 will be described hereinafter. Selected locations W, whichwere selected prior to the flight, and an actual flight route obtainedfrom the log data, are shown on the right side in FIG. 9. On the otherhand, a still image (expected to having been photographed at thelocation) corresponding to the location, which is selected in the flightroute on the right side by the cloverleaf-shaped marker, is shown on theleft side in FIG. 9.

An actual flight route obtained from the log data is shown on the rightside in FIG. 10. Here, the arrow marked points correspond to thelocations where the still images were photographed, respectively. Whenthe operator selects the arrow mark, the selected point is changed intoa cloverleaf-shaped marker, and a corresponding still image is shown onthe left side. At this time, an arrow for indicating the direction(flight direction) of the UAV is shown in the still image on the leftside. The same processing is also performed in the case shown in FIGS.11 to 14.

An example of a processing procedure relating to FIGS. 10 to 14 will bedescribed below. Here, FIG. 10 is described as a typical example. FIG. 7shows an example of the processing procedure in this case. First, as apreparation, the processing shown in FIG. 6 is performed, and the imagedata and the log data are synchronized (matched).

When the operator selects a specific photographing location (specificarrow marked position) of the flight route on the right side in the GUIimage in FIG. 10 (step S111), a corresponding still image is obtained(step S112). The processing of step S111 is performed by the route pointselection receiving unit 121 shown in FIG. 3, and the processing of stepS112 is performed by the corresponding still image obtaining unit 122shown in FIG. 3.

After step S112, the procedure advances to step S113, and a processingfor embedding an arrow (moving-direction indication) in the still imageobtained in step S112 for indicating the flight direction of the UAV isperformed. This processing is performed by the moving-directionindication embedding unit 123 shown in FIG. 3. Next, the procedureadvances to step S114, and a processing for displaying the still image,in which the arrow (moving-direction indication) for indicating theflight direction of the UAV is embedded, on the display 104 shown inFIG. 1. This processing is performed by the display controlling unit 124shown in FIG. 3.

Although it depends on the performances of the GNSS unit and the IMUequipped on the UAV, the log data obtained by the UAV contains a marginof error in most cases. In particular, when a device of low cost type isused, this tendency increases. In addition, the interval of obtainingthe location information by the UAV does not necessarily coincide withthe photographing interval of the camera. When general-use products areused together, there may be cases in which the time information of thelog data differs from the time information of the image data, in thefirst place. Therefore, there may be cases in which the locationselected by the operator does not correspond to the still image shown onthe left side in FIG. 10. That is, there is a possibility that thematching is not perfectly performed, and therefore, a still image, whichis different from a still image photographed at the selected location byseveral frames (a still image photographed at a location that isslightly different from the selected location), may be selected anddisplayed.

By using the images shown in FIGS. 10 to 14, such a frame shift isdetected. In this case, the shift can be corrected manually by theoperator. The detection of the shift is performed as follows. As shownin FIGS. 10 to 14, the operator selects a flight location along theflight route and confirms the still image shown on the left side at thesame time. At this time, an arrow for indicating the flight direction ofthe UAV is shown in the still image. In addition, a moving distance fromthe location, at which a preceding still image was photographed, isadded to the arrow.

By looking at the changes from FIG. 11 to FIG. 12 and then from FIG. 12to FIG. 13, change in the direction of the arrow shown in the stillimage according to the turn of the UAV is observed. By comparing thechange in the route on the right side with the change in the directionof the arrow in the still image on the left side at turning, thecorrectness of the corresponding relationship between the flight routeand the still image is examined. For example, if the direction of theUAV is clearly changed in the flight route on the right side, but nochange in the direction of the arrow can be recognized in the stillimage on the left side, between two timings that are consecutive orclose to each other, mismatch of the relationship between the flightroute and the still image is known. In addition, the mismatcheddirection is known from the difference between the timings when thechange appears in the flight route and the still image, respectively.

For example, when the GUI images shown in FIGS. 11 to 14 are displayedin the time sequence order, a change in the direction of the arrow inthe still image on the left side may be observed slightly after a changein the direction of the UAV is observed in the flight route on the rightside. In this case, a still image, which is photographed at a locationsubsequent to the actual location in the fight route, is made tocorrespond to the flight route incorrectly. In this case, by shiftingthe corresponding relationship by the corresponding amount of the timeshift, a corrected corresponding relationship between the image data andthe log data is obtained.

The above operation can be visually performed by using the GUI. Ifcostly devices are used, data, in which the corresponding relationshipbetween the image data and the log data is specified correctly, may beobtained. However, a camera, a GNSS unit, and an IMU, of high costtypes, are not only expensive, but are large in dimensions, in weight,and in electric power consumption. As a result, an expensive and largeUAV is required correspondingly, which is undesirable.

By using the technique of this embodiment, the image data and the logdata can be synchronized with each other even when hardware of a lowcost type is used. Moreover, even when there is a shift in thesynchronized data, the shift can be corrected by using the GUI images asshown in FIGS. 10 to 14 by easy operation.

What is claimed is:
 1. A survey data processing device comprising: aninput unit configured to receive image data of multiple still images,which are photographed from a mobile body flying, and receive flightdata, in which a flight route of the mobile body is measured; a flightroute estimating unit configured to estimate a flight route of themobile body based on changes in positions of feature points included inthe multiple still images on a screen; and a matching relationshipspecifying unit configured to specify a matching relationship betweenthe flight route of the flight data and the estimated flight route. 2.The survey data processing device according to claim 1, wherein theflight route estimating unit is configured to perform a processing forobtaining a moving-direction vector, which characterizes a movement ofthe feature point on the screen, and a processing for estimating theflight route based on the moving-direction vector.
 3. The survey dataprocessing device according to claim 1, wherein the matchingrelationship specifying unit is configured to calculate a location ofthe estimated flight route so that a total of distances between vectorsfor indicating the estimated flight route and vectors for indicating theflight route of the flight data will be minimum.
 4. A survey dataprocessing method comprising: receiving image data of multiple stillimages, which are photographed from a mobile body flying, and flightdata, in which a flight route of the mobile body is measured; estimatinga flight route of the mobile body based on changes in positions offeature points included in the multiple still images on a screen; andspecifying a matching relationship between the flight route of theflight data and the estimated flight route.
 5. A computer programproduct comprising a non-transitory computer-readable medium storingcomputer-executable program codes, the computer-executable program codescomprising program code instructions for: receiving image data ofmultiple still images, which are photographed from a mobile body flying,and flight data, in which a flight route of the mobile body is measured;estimating a flight route of the mobile body based on changes inpositions of feature points included in the multiple still images on ascreen; and specifying a matching relationship between the flight routeof the flight data and the estimated flight route.
 6. The survey dataprocessing device according to claim 2, wherein the matchingrelationship specifying unit is configured to calculate a location ofthe estimated flight route so that a total of distances between vectorsfor indicating the estimated flight route and vectors for indicating theflight route of the flight data will be minimum.